Proximity Based Noise and Chat

ABSTRACT

Disclosed are systems, methods, and non-transitory computer-readable storage media for altering and combining real and simulated audio streams. For example, a system can determine a location of a first unmanned aerial vehicle (UAV). The system can then determine a location of an object and can receive an audio stream associated with the object. The system can then determine a distance between the location of the first UAV and the location of the object. The system can adjust the audio stream volume according to the distance. The system can then send the audio stream to a listener.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation and claims the prioritybenefit of U.S. patent application Ser. No. 15/394,313 filed Dec. 29,2016, now U.S. Pat. No. 10,067,736, which claims the priority benefit ofU.S. patent application 62/402,721 filed Sep. 30, 2016, the disclosuresof which are incorporated herein by reference.

BACKGROUND Field of the Invention

The present technology pertains to combining real and simulated audiostreams, and more specifically pertains to modifying and combining audiostreams based on simulated and actual locations of simulated and realobjects.

Description of the Related Art

Augmented reality has recently become popular with the proliferation ofpowerful cellular phones. Games and software are overlaying video feedswith simulated creatures, information, or other objects. However, thesesimply overlays lack the immersion that some users expect. For example,they generally only use video which can be impressive but not completelyimmersive. Furthermore, real and simulated objects rarely interact.

SUMMARY OF THE CLAIMED INVENTION

Additional features and advantages of the disclosure will be set forthin the description which follows, and in part will be obvious from thedescription, or can be learned by practice of the herein disclosedprinciples. The features and advantages of the disclosure can berealized and obtained by means of the instruments and combinationsparticularly pointed out in the appended claims. These and otherfeatures of the disclosure will become more fully apparent from thefollowing description and appended claims, or can be learned by thepractice of the principles set forth herein.

Disclosed are systems, methods, and non-transitory computer-readablestorage media for altering and combining real and simulated audiostreams. For example, a system can determine a location of a firstunmanned aerial vehicle (UAV). The system can then determine a locationof an object and can receive an audio stream associated with the object.The system can then determine a distance between the location of thefirst UAV and the location of the object. The system can adjust theaudio stream volume according to the distance. The system can then sendthe audio stream to a listener.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features of the disclosure can be obtained, a moreparticular description of the principles briefly described above will berendered by reference to specific embodiments thereof which areillustrated in the appended drawings. Understanding that these drawingsdepict only exemplary embodiments of the disclosure and are nottherefore to be considered to be limiting of its scope, the principlesherein are described and explained with additional specificity anddetail through the use of the accompanying drawings in which:

FIG. 1 shows an unmanned aerial vehicle (UAV) according to someembodiments;

FIG. 2 shows a control transmitter according to some embodiments;

FIG. 3 shows a display according to some embodiments;

FIG. 4 shows various real and simulated objects in an environment;

FIG. 5 shows a UAV with an array of directional microphones according tosome embodiments; and

FIG. 6A and FIG. 6B illustrate example system embodiments.

DETAILED DESCRIPTION

Various embodiments of the disclosure are discussed in detail below.While specific implementations are discussed, it should be understoodthat this is done for illustration purposes only. A person skilled inthe relevant art will recognize that other components and configurationsmay be used without parting from the spirit and scope of the disclosure.

The disclosed technology addresses the need in the art for modifying andcombining audio streams based on simulated and actual locations ofsimulated and real objects.

FIG. 1 shows unmanned aerial vehicle (UAV) 100 according to someembodiments. UAV 100 can have one or more motors 150 configured torotate attached propellers 155 in order to control the position of UAV100 in the air. UAV 100 can be configured as a fixed wing vehicle (e.g.,airplane), a rotary vehicle (e.g., a helicopter or multirotor), or ablend of the two. For the purpose of FIG. 1, axes 175 can assist in thedescription of certain features. If UAV 100 is oriented parallel to theground, the Z axis can be the axis perpendicular to the ground, the Xaxis can generally be the axis that passes through the bow and stern ofUAV 100, and the Y axis can be the axis that pass through the port andstarboard sides of UAV 100. Axes 175 are merely provided for convenienceof the description herein.

In some embodiments, UAV 100 has main body 110 with one or more arms140. The proximal end of arm 140 can attach to main body 110 while thedistal end of arm 140 can secure motor 150. Arms 140 can be secured tomain body 110 in an “X” configuration, an “H” configuration, a “T”configuration, or any other configuration as appropriate. The number ofmotors 150 can vary, for example there can be three motors 150 (e.g., a“tricopter”), four motors 150 (e.g., a “quadcopter”), eight motors(e.g., an “octocopter”), etc.

In some embodiments, each motor 155 rotates (i.e., the drive shaft ofmotor 155 spins) about parallel axes. For example, the thrust providedby all propellers 155 can be in the Z direction. Alternatively, a motor155 can rotate about an axis that is perpendicular (or any angle that isnot parallel) to the axis of rotation of another motor 155. For example,two motors 155 can be oriented to provide thrust in the Z direction(e.g., to be used in takeoff and landing) while two motors 155 can beoriented to provide thrust in the X direction (e.g., for normal flight).In some embodiments, UAV 100 can dynamically adjust the orientation ofone or more of its motors 150 for vectored thrust.

In some embodiments, the rotation of motors 150 can be configured tocreate or minimize gyroscopic forces. For example, if there are an evennumber of motors 150, then half of the motors can be configured torotate counter-clockwise while the other half can be configured torotate clockwise. Alternating the placement of clockwise andcounter-clockwise motors can increase stability and enable UAV 100 torotate about the z-axis by providing more power to one set of motors 150(e.g., those that rotate clockwise) while providing less power to theremaining motors (e.g., those that rotate counter-clockwise).

Motors 150 can be any combination of electric motors, internalcombustion engines, turbines, rockets, etc. In some embodiments, asingle motor 150 can drive multiple thrust components (e.g., propellers155) on different parts of UAV 100 using chains, cables, gearassemblies, hydraulics, tubing (e.g., to guide an exhaust stream usedfor thrust), etc. to transfer the power.

In some embodiments, motor 150 is a brushless motor and can be connectedto electronic speed controller X45. Electronic speed controller 145 candetermine the orientation of magnets attached to a drive shaft withinmotor 150 and, based on the orientation, power electromagnets withinmotor 150. For example, electronic speed controller 145 can have threewires connected to motor 150, and electronic speed controller 145 canprovide three phases of power to the electromagnets to spin the driveshaft in motor 150. Electronic speed controller 145 can determine theorientation of the drive shaft based on back-emf on the wires or bydirectly sensing to position of the drive shaft.

Transceiver 165 can receive control signals from a control unit (e.g., ahandheld control transmitter, a server, etc.). Transceiver 165 canreceive the control signals directly from the control unit or through anetwork (e.g., a satellite, cellular, mesh, etc.). The control signalscan be encrypted. In some embodiments, the control signals includemultiple channels of data (e.g., “pitch,” “yaw,” “roll,” “throttle,” andauxiliary channels). The channels can be encoded usingpulse-width-modulation or can be digital signals. In some embodiments,the control signals are received over TC/IP or similar networking stack.

In some embodiments, transceiver 165 can also transmit data to a controlunit. Transceiver 165 can communicate with the control unit usinglasers, light, ultrasonic, infra-red, Bluetooth, 602.11x, or similarcommunication methods, including a combination of methods. Transceivercan communicate with multiple control units at a time.

Position sensor 135 can include an inertial measurement unit fordetermining the acceleration and/or the angular rate of UAV 100, a GPSreceiver for determining the geolocation and altitude of UAV 100, amagnetometer for determining the surrounding magnetic fields of UAV 100(for informing the heading and orientation of UAV 100), a barometer fordetermining the altitude of UAV 100, etc. Position sensor 135 caninclude a land-speed sensor, an air-speed sensor, a celestial navigationsensor, etc.

UAV 100 can have one or more environmental awareness sensors. Thesesensors can use sonar, LiDAR, stereoscopic imaging, computer vision,etc. to detect obstacles and determine the nearby environment. Forexample, a collision avoidance system can use environmental awarenesssensors to determine how far away an obstacle is and, if necessary,change course.

Position sensor 135 and environmental awareness sensors can all be oneunit or a collection of units. In some embodiments, some features ofposition sensor 135 and/or the environmental awareness sensors areembedded within flight controller 130.

In some embodiments, an environmental awareness system can take inputsfrom position sensors 135, environmental awareness sensors, databases(e.g., a predefined mapping of a region) to determine the location ofUAV 100, obstacles, and pathways. In some embodiments, thisenvironmental awareness system is located entirely on UAV 100,alternatively, some data processing can be performed external to UAV100.

Camera 105 can include an image sensor (e.g., a CCD sensor, a CMOSsensor, etc.), a lens system, a processor, etc. The lens system caninclude multiple movable lenses that can be adjusted to manipulate thefocal length and/or field of view (i.e., zoom) of the lens system. Insome embodiments, camera 105 is part of a camera system which includesmultiple cameras 105. For example, two cameras 105 can be used forstereoscopic imaging (e.g., for first person video, augmented reality,etc.). Another example includes one camera 105 that is optimized fordetecting hue and saturation information and a second camera 105 that isoptimized for detecting intensity information. In some embodiments,camera 105 optimized for low latency is used for control systems while acamera 105 optimized for quality is used for recording a video (e.g., acinematic video). Camera 105 can be a visual light camera, an infraredcamera, a depth camera, etc.

A gimbal and dampeners can help stabilize camera 105 and remove erraticrotations and translations of UAV 100. For example, a three-axis gimbalcan have three stepper motors that are positioned based on a gyroscopereading in order to prevent erratic spinning and/or keep camera 105level with the ground.

Video processor 125 can process a video signal from camera 105. Forexample video process 125 can enhance the image of the video signal,down-sample or up-sample the resolution of the video signal, add audio(captured by a microphone) to the video signal, overlay information(e.g., flight data from flight controller 130 and/or position sensor),convert the signal between forms or formats, etc.

Video transmitter 120 can receive a video signal from video processor125 and transmit it using an attached antenna. The antenna can be acloverleaf antenna or a linear antenna. In some embodiments, videotransmitter 120 uses a different frequency or band than transceiver 165.In some embodiments, video transmitter 120 and transceiver 165 are partof a single transceiver.

Battery 170 can supply power to the components of UAV 100. A batteryelimination circuit can convert the voltage from battery 170 to adesired voltage (e.g., convert 12v from battery 170 to 5v for flightcontroller 130). A battery elimination circuit can also filter the powerin order to minimize noise in the power lines (e.g., to preventinterference in transceiver 165 and transceiver 120). Electronic speedcontroller 145 can contain a battery elimination circuit. For example,battery 170 can supply 12 volts to electronic speed controller 145 whichcan then provide 5 volts to flight controller 130. In some embodiments,a power distribution board can allow each electronic speed controller(and other devices) to connect directly to the battery.

In some embodiments, battery 170 is a multi-cell (e.g., 2S, 3S, 4S,etc.) lithium polymer battery. Battery 170 can also be a lithium-ion,lead-acid, nickel-cadmium, or alkaline battery. Other battery types andvariants can be used as known in the art. Additional or alternative tobattery 170, other energy sources can be used. For example, UAV 100 canuse solar panels, wireless power transfer, a tethered power cable (e.g.,from a ground station or another UAV 100), etc. In some embodiments, theother energy source can be utilized to charge battery 170 while inflight or on the ground.

Battery 170 can be securely mounted to main body 110. Alternatively,battery 170 can have a release mechanism. In some embodiments, battery170 can be automatically replaced. For example, UAV 100 can land on adocking station and the docking station can automatically remove adischarged battery 170 and insert a charged battery 170. In someembodiments, UAV 100 can pass through docking station and replacebattery 170 without stopping.

Battery 170 can include a temperature sensor for overload prevention.For example, when charging, the rate of charge can be thermally limited(the rate will decrease if the temperature exceeds a certain threshold).Similarly, the power delivery at electronic speed controllers 145 can bethermally limited—providing less power when the temperature exceeds acertain threshold. Battery 170 can include a charging and voltageprotection circuit to safely charge battery 170 and prevent its voltagefrom going above or below a certain range.

UAV 100 can include a location transponder. For example, in a racingenvironment, race officials can track UAV 100 using locationtransponder. The actual location (e.g., X, Y, and Z) can be trackedusing triangulation of the transponder. In some embodiments, gates orsensors in a track can determine if the location transponder has passedby or through the sensor or gate.

Flight controller 130 can communicate with electronic speed controller145, battery 170, transceiver 165, video processor 125, position sensor135, and/or any other component of UAV 100. In some embodiments, flightcontroller 130 can receive various inputs (including historical data)and calculate current flight characteristics. Flight characteristics caninclude an actual or predicted position, orientation, velocity, angularmomentum, acceleration, battery capacity, temperature, etc. of UAV 100.Flight controller 130 can then take the control signals from transceiver165 and calculate target flight characteristics. For example, targetflight characteristics might include “rotate x degrees” or “go to thisGPS location”. Flight controller 130 can calculate responsecharacteristics of UAV 100. Response characteristics can include howelectronic speed controller 145, motor 150, propeller 155, etc. respond,or are expected to respond, to control signals from flight controller130. Response characteristics can include an expectation for how UAV 100as a system will respond to control signals from flight controller 130.For example, response characteristics can include a determination thatone motor 150 is slightly weaker than other motors.

After calculating current flight characteristics, target flightcharacteristics, and response characteristics flight controller 130 cancalculate optimized control signals to achieve the target flightcharacteristics. Various control systems can be implemented during thesecalculations. For example a proportional-integral-derivative (PID) canbe used. In some embodiments, an open-loop control system (i.e., onethat ignores current flight characteristics) can be used. In someembodiments, some of the functions of flight controller 130 areperformed by a system external to UAV 100. For example, current flightcharacteristics can be sent to a server that returns the optimizedcontrol signals. Flight controller 130 can send the optimized controlsignals to electronic speed controllers 145 to control UAV 100.

In some embodiments, UAV 100 has various outputs that are not part ofthe flight control system. For example, UAV 100 can have a loudspeakerfor communicating with people or other UAVs 100. Similarly, UAV 100 canhave a flashlight or laser. The laser can be used to “tag” another UAV100.

UAV 100 can have blaster 115 for interacting with other UAVs, theenvironment, or simulated objects. Blaster 115 can be a laser,flashlight (e.g., for infrared or visible light), projectile launcher(e.g., to send darts, balls, or other objects in the air), etc. In someembodiments, blaster 115 is passive and merely serves as an indicatorfor the positioning of a corresponding simulated blaster that

FIG. 2 shows control transmitter 200 according to some embodiments.Control transmitter 200 can send control signals to transceiver 165.Control transmitter can have auxiliary switches 210, joysticks 215 and220, and antenna 205. Joystick 215 can be configured to send elevatorand aileron control signals while joystick 220 can be configured to sendthrottle and rudder control signals (this is termed a mode 2configuration). Alternatively, joystick 215 can be configured to sendthrottle and aileron control signals while joystick 220 can beconfigured to send elevator and rudder control signals (this is termed amode 1 configuration). Auxiliary switches 210 can be configured to setoptions on control transmitter 200 or UAV 100. In some embodiments,control transmitter 200 receives information from a transceiver on UAV100. For example, it can receive some current flight characteristicsfrom UAV 100.

FIG. 3 shows display 300 according to some embodiments. Display 300 caninclude battery 305 or another power source, display screen 310, andreceiver 315. Display 300 can receive a video stream from transmitter120 from UAV 100. Display 300 can be a head-mounted unit as depicted inFIG. 3. Display 300 can be a monitor such that multiple viewers can viewa single screen. In some embodiments, display screen 310 includes twoscreens, one for each eye; these screens can have separate signals forstereoscopic viewing. In some embodiments, receiver 315 is mounted ondisplay 3100 (as shown in FIG. 3), alternatively, receiver 315 can be aseparate unit that is connected using a wire to display 300. In someembodiments, display 300 is mounted on control transmitter 200.

FIG. 4 shows various real and simulated objects in environment 400.Environment 400 can be a map of a physical location (e.g., a park, afield, a building, a region, etc.) or a simulated location. Environment400 is shown in FIG. 4 with a grid to indicate the positions of objectswithin environment 400. Environment 400 can contain real objects suchas: UAVs 100 _(a), 100 _(b), 100 _(c), and 100 _(d); controltransmitters 200 _(a) and 200 _(b); obstruction 410; sensors 425 _(a),425 _(b), and 425 _(c); base station 420; etc. Environment 400 cancontain simulated objects such as sound emitters 405 _(a) and 405 _(b)as well as obstruction 415. In some embodiments, the locations of thereal objects can be mapped into a simulated space with the simulatedobjects (e.g., projecting the real objects into a simulated space).Alternatively, the locations of the simulated objects can be mapped intoa real-space with the real objects (e.g., projecting the simulatedobjects into a real space). In some embodiments, absolute positions areunknown and only relative positions between real and simulated objectscan be calculated.

In some embodiments, a real object can represent a simulated object. Forexample, a real balloon can represent the position of a simulateddragon. Objects that have a physical presence can be termed “real” whileobjects that lack a physical presence can be termed “simulated.”

Control transmitter 200 _(a) can be connected to UAV 100 _(a) whiletransmitter 200 _(b) can be connected to UAV 100 _(b). An operator canoperate each control transmitter 200 to control UAV 100.

Base station 420 can be in communication with real objects inenvironment 400, for example base station 420 can wirelessly communicate(using, e.g., Bluetooth, infrared, wifi, light, etc.) with UAV 100 _(a)and gather telemetry, send instructions, etc. Base station can, usingsensors 425 _(a), 425 _(b), and 425 _(c), detect the positions of realobjects in environment 400. Position can include location, orientation,and configuration (e.g., the shape and makeup of the object). Sensors425 _(a), 425 _(b), and 425 _(c) can use visible light cameras, infraredcameras, LIDAR, RADAR, or other similar systems to detect the positionof objects. In some embodiments, sensors 425 _(a), 425 _(b), and 425_(c) detect a marker (visible or RF) on objects to determine theirposition. A marker on objects can be passive or actively powered.

In some embodiments, base station 420 receives signals from controltransmitters 200 and, after processing them if necessary, sends controlsignals to UAVs 100. In some embodiments, some UAVs are controlled by anoperator (e.g., UAV 100 _(a) and 100 _(b)) while others can beautonomously controlled (e.g., UAV 100 _(c) and UAV 100 _(d)) by basestation 420.

Base station 420 can act as an arena server and can be a centrallocation for data processing, control signals, video signals, audiosignals, etc. Base station 420 can keep track of real and simulatedobjects. Base station 420 can simulate the actions of simulated objects.For example, a simulated object can be a simulated enemy fighter planeand base station 420 can have artificial intelligence that dictates theactions of the simulated enemy fighter plane.

Base station 420 can simulate environment 400. For example, environmentcan be representative of a real location such as a park with hills,benches, and trees. Base station 420 can determine the topology of thepark and create simulated representations of the hills, benches, andtrees (e.g., using sensors 425 _(a), 425 _(b), and 425 _(c), usinginformation from a database, etc.). Base station 420 can simulate anenvironment that takes few or no cues from the actual location ofenvironment 400. For example, base station 420 can simulate an outerspace environment even though environment 400 is representative of apark.

Base station 420 can simulate atmospheric and sensory aspects ofenvironment 400. For example, base station 420 can renderthree-dimensional views of environment 420 from various locations (e.g.,at the location of UAV 100 _(a) to provide a simulated first-person viewfrom UAV 100 _(a)).

Base station 420 can facilitate the playing of games within environment400. For example, base station 420 can simulate and facilitate gameslike capture the flag, racing, tag, dogfighting, etc.

In some embodiments, base station 420 can receive multiple audio streams(e.g., recorded at microphones associated with real objects such ascontrol transmitter 200 _(a), control transmitter 200 _(b), UAV 100_(a), UAV 100 _(b), etc. Base station 420 can generate audio streams forreal and/or simulated objects. For example, if simulated object 405 _(b)is a volcano, base station 420 can generate an audio stream of thevolcano erupting and associate the audio stream with the location ofsimulated object 405 _(b). Base station 420 can then filter, alter, andrecombine the audio streams for presentation. For example, an operatorthat is controlling UAV 100 _(a) with control transmitter 200 _(a) canwear headphones (or have a speaker) connected to base station 420. Basestation 420 can generate an output audio stream using real and simulatedaudio streams. This output audio stream can be associated with thelocation of UAV 100 _(a) but can be sent to the operator's headphones.

Base station 420 can determine the distance between UAV 100 _(a) andnearby sound emitters (e.g., sound sources such as real or simulatedobjects). It can then combine the audio streams to generate the outputaudio stream. Nearby objects (whether real or simulated) can have theirassociated audio streams amplified while far objects can have theirassociated audio streams weakened (or diminished in volume). Forexample, an audio stream from UAV 100 _(b) can be loud while an audiostream from UAV 100 _(d) can be softer. In some embodiments, a delay canbe applied proportional to the distance to simulate the travel time ofthe audio stream.

In some embodiments, an audio stream can be occluded (or blocked). Forexample, UAV 100 _(b) can be on one side of obstruction 410 while UAV100 _(a) can be on the other side of obstruction 410. Becauseobstruction 410 is a real object, it might occlude sound from UAV 100_(b) such that the sound is altered (e.g., softened or distorted) by thetime it reaches UAV 100 _(a). Base station 420 can similarly alter anaudio stream associated with UAV 100 _(b) as it is sent to a listenerassociated with UAV 100 _(a). For example, an operator at controltransmitter 200 _(b) can speak into a microphone which can then send anaudio stream to base station 420. Base station 420 can determine thatthe audio steam is associated with UAV 100 _(b) and that a listener(e.g., an operator at control transmitter 200 _(a)) is associated withUAV 100 _(a). Base station 420 can then alter the audio stream and sendit to the listener (unless the occlusion completely blocks the audiostream).

Objects (real and simulated) can have occlusion properties. For example,an object might block or degrade certain frequencies. In someembodiments, a real object has certain real occlusions properties whilebase station 420 can apply simulated occlusion properties to the object.For example, a real wall might block most sound that passes through itbut base station 420 can simulate the wall having other properties(e.g., it might not block sound at all).

In some embodiments, an audio stream can be reflected. For example, whencreating an output audio stream for UAV 100 _(b), base station 420 cancombine a direct audio stream from simulated object 405 _(a) with areflected audio stream from simulated object 405 _(a) as if it werereflected off obstruction 410. The reflected audio stream can be delayed(relative to the direct audio stream) based on the greater distance oftravel. The reflected audio stream can have a decreased volume based onthe travel. The reflected audio stream can have certain frequenciespartially absorbed based on the characteristics of obstruction 410. Forexample, a cloth obstruction can absorb high frequency sounds while anobstruction with gaps can reflect primarily high frequency sounds. Ahigh pass or low pass filter can simulate these absorptive properties.In some embodiments, multiple reflected audio streams can be generated;e.g., a reflected audio stream off obstruction 410, and another off theground.

In some embodiments, base station 420 can simulate a wind. For example,if a simulated wind is moving from top right to bottom left in FIG. 4,base station 420 can make an audio stream associated with UAV 100 _(d)be louder to an output audio stream for UAV 100 _(a) while an audiostream associated with UAV 100 _(b) can be softer. The simulated windcan be based on actual wind existing in environment 400. For example, awind-speed device can detect the speed of the actual wind.

In some embodiments, base station 420 can simulate Doppler shifts. Forexample, an audio stream associated with a (real or simulated) objectmoving towards or away from a reference object can be shifted to behigher or lower pitched based on the motion.

In some embodiments, real obstructions are dynamically discovered. Forexample, UAV 100 _(a) can send a test signal (e.g., an ultrasonic orinfrared pulse) which UAV 100 _(b) can detect. The test signal can havemultiple component signals (e.g., multiple frequencies). Based on theintensity and quality of the received test signal, base station 420 candetermine appropriate alterations that can be made to emulate theocclusion. For example, if the test signal is emitted at one frequency,but detected at a different frequency, the frequency shift can bereplicated with the audio stream associated with UAV 100 _(a). Otheralterations to a sound can be detected. For example, reflections (e.g.,an echo), interference, equalization (e.g., enhanced or diminishedintensity of various component frequencies), Doppler shifts, etc.

In some embodiments, various objects can be part of groupings. Therespective audio streams of objects can be active, muted, or alteredbased on their groupings. For example, if objects are organized intoteams, output audio streams can include only audio streams associatedwith the object's team. For example, UAV 100 _(a) and an operator ofcontrol transmitter 200 _(a) can be on a team with UAV 100 _(b) and itsassociated operator of control transmitter 200 _(b). An audio streamassociated with UAV 100 _(b) and an audio stream associated with controltransmitter 200 _(b) can be combined into an output audio stream for theoperator of control transmitter 200 _(a). If UAV 100 _(c) and UAV 100_(d) are on a different team, their associated audio streams can beisolated from the output audio stream for the operator of controltransmitter 200 _(a). In some embodiments, objects can be parented suchthat a “parent” can receive audio streams from its children, but itschildren can only receive audio streams from the parent (and notsiblings). This can be useful if one operator is assigned as a captainfor a team of other operators.

FIG. 5 shows UAV 100 _(c) with an array of directional microphonesaccording to some embodiments. UAV 100 _(c) can have an array ofdirectional microphones (e.g., microphones 505 _(a)-505 _(h)), each witha respective recording region. There can be any number of directionalmicrophones (e.g., one, two, four, eight, etc.). The directionalmicrophones can be arranged in a circle (e.g., they can all becoplanar); alternatively, some microphones can be directed above andbelow the other microphones for spherical coverage. Base station 420 canuse position data for objects (e.g., UAV 100 _(f)) to determine whichaudio streams should be active. For example, when UAV 100 _(f) is inlocation 1, the audio stream from microphone 505 _(f) can be active(e.g., provided to an output audio stream associated with UAV 100 _(c)).As UAV 100 _(f) moves away from location 1, the audio stream frommicrophone 505 _(f) can be gradually muted. As UAV 100 _(f) moves intolocation 2, the audio stream from microphone 505 _(c) can be graduallyactivated (e.g., provided to an output audio stream associated with UAV100 _(c)).

As an object (e.g., UAV 100 _(f)) moves “behind” a simulated obstruction415, the respective audio stream can be modified accordingly. Forexample, even though microphone 505 _(c) can detect sound from UAV 100_(f), base station 420 can determine that simulated obstruction 415 isin between UAV 100 _(c) and UAV 100 _(f) at location 2. Base station canthen decrease the volume of the audio stream from microphone 505 _(c) orotherwise modify the audio stream based on audio occlusion properties ofsimulated obstruction 415.

In some embodiments multiple microphones 505 can be placed a distanceapart and can isolate audio streams based on the time it takes thestream to get to the respective microphones.

It should be understood that multiple audio streams from various sources(real or simulated) can be mapped to environment 400, modified accordingoccluding and/or reflecting objects (real or simulated), combinedaccording to a reference location, and sent to a listener (who may be ata different location than reference location).

FIG. 6A and FIG. 6B illustrate example system embodiments. The moreappropriate embodiment will be apparent to those of ordinary skill inthe art when practicing the present technology. Persons of ordinaryskill in the art will also readily appreciate that other systemembodiments are possible.

FIG. 6A illustrates a conventional system bus computing systemarchitecture 600 wherein the components of the system are in electricalcommunication with each other using a bus 605. Exemplary system 600includes a processing unit (CPU or processor) 610 and a system bus 605that couples various system components including the system memory 615,such as read only memory (ROM) 670 and random access memory (RAM) 675,to the processor 610. The system 600 can include a cache of high-speedmemory connected directly with, in close proximity to, or integrated aspart of the processor 610. The system 600 can copy data from the memory615 and/or the storage device 630 to the cache 612 for quick access bythe processor 610. In this way, the cache can provide a performanceboost that avoids processor 610 delays while waiting for data. These andother modules can control or be configured to control the processor 610to perform various actions. Other system memory 615 may be available foruse as well. The memory 615 can include multiple different types ofmemory with different performance characteristics. The processor 610 caninclude any general purpose processor and a hardware module or softwaremodule, such as module 1 637, module 7 634, and module 3 636 stored instorage device 630, configured to control the processor 910 as well as aspecial-purpose processor where software instructions are incorporatedinto the actual processor design. The processor 610 may essentially be acompletely self-contained computing system, containing multiple cores orprocessors, a bus, memory controller, cache, etc. A multi-core processormay be symmetric or asymmetric.

To enable user interaction with the computing device 600, an inputdevice 645 can represent any number of input mechanisms, such as amicrophone for speech, a touch-sensitive screen for gesture or graphicalinput, keyboard, mouse, motion input, speech and so forth. An outputdevice 635 can also be one or more of a number of output mechanismsknown to those of skill in the art. In some instances, multimodalsystems can enable a user to provide multiple types of input tocommunicate with the computing device 600. The communications interface640 can generally govern and manage the user input and system output.There is no restriction on operating on any particular hardwarearrangement and therefore the basic features here may easily besubstituted for improved hardware or firmware arrangements as they aredeveloped.

Storage device 630 is a non-volatile memory and can be a hard disk orother types of computer readable media which can store data that areaccessible by a computer, such as magnetic cassettes, flash memorycards, solid state memory devices, digital versatile disks, cartridges,random access memories (RAMs) 675, read only memory (ROM) 670, andhybrids thereof.

The storage device 630 can include software modules 637, 634, 636 forcontrolling the processor 610. Other hardware or software modules arecontemplated. The storage device 630 can be connected to the system bus605. In one aspect, a hardware module that performs a particularfunction can include the software component stored in acomputer-readable medium in connection with the necessary hardwarecomponents, such as the processor 610, bus 605, display 635, and soforth, to carry out the function.

FIG. 6B illustrates an example computer system 650 having a chipsetarchitecture that can be used in executing the described method andgenerating and displaying a graphical user interface (GUI). Computersystem 650 is an example of computer hardware, software, and firmwarethat can be used to implement the disclosed technology. System 650 caninclude a processor 655, representative of any number of physicallyand/or logically distinct resources capable of executing software,firmware, and hardware configured to perform identified computations.Processor 655 can communicate with a chipset 660 that can control inputto and output from processor 655. In this example, chipset 660 outputsinformation to output 665, such as a display, and can read and writeinformation to storage device 670, which can include magnetic media, andsolid state media, for example. Chipset 660 can also read data from andwrite data to RAM 675. A bridge 680 for interfacing with a variety ofuser interface components 685 can be provided for interfacing withchipset 660. Such user interface components 685 can include a keyboard,a microphone, touch detection and processing circuitry, a pointingdevice, such as a mouse, and so on. In general, inputs to system 650 cancome from any of a variety of sources, machine generated and/or humangenerated.

Chipset 660 can also interface with one or more communication interfaces690 that can have different physical interfaces. Such communicationinterfaces can include interfaces for wired and wireless local areanetworks, for broadband wireless networks, as well as personal areanetworks. Some applications of the methods for generating, displaying,and using the GUI disclosed herein can include receiving ordereddatasets over the physical interface or be generated by the machineitself by processor 655 analyzing data stored in storage 670 or 675.Further, the machine can receive inputs from a user via user interfacecomponents 685 and execute appropriate functions, such as browsingfunctions by interpreting these inputs using processor 655.

It can be appreciated that example systems 600 and 650 can have morethan one processor 610 or be part of a group or cluster of computingdevices networked together to provide greater processing capability.

For clarity of explanation, in some instances the present technology maybe presented as including individual functional blocks includingfunctional blocks comprising devices, device components, steps orroutines in a method embodied in software, or combinations of hardwareand software.

In some embodiments the computer-readable storage devices, mediums, andmemories can include a cable or wireless signal containing a bit streamand the like. However, when mentioned, non-transitory computer-readablestorage media expressly exclude media such as energy, carrier signals,electromagnetic waves, and signals per se.

Methods according to the above-described examples can be implementedusing computer-executable instructions that are stored or otherwiseavailable from computer readable media. Such instructions can comprise,for example, instructions and data which cause or otherwise configure ageneral purpose computer, special purpose computer, or special purposeprocessing device to perform a certain function or group of functions.Portions of computer resources used can be accessible over a network.The computer executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, firmware, orsource code. Examples of computer-readable media that may be used tostore instructions, information used, and/or information created duringmethods according to described examples include magnetic or opticaldisks, flash memory, USB devices provided with non-volatile memory,networked storage devices, and so on.

Devices implementing methods according to these disclosures can comprisehardware, firmware and/or software, and can take any of a variety ofform factors. Typical examples of such form factors include laptops,smart phones, small form factor personal computers, personal digitalassistants, rackmount devices, standalone devices, and so on.Functionality described herein also can be embodied in peripherals oradd-in cards. Such functionality can also be implemented on a circuitboard among different chips or different processes executing in a singledevice, by way of further example.

The instructions, media for conveying such instructions, computingresources for executing them, and other structures for supporting suchcomputing resources are means for providing the functions described inthese disclosures.

Although a variety of examples and other information was used to explainaspects within the scope of the appended claims, no limitation of theclaims should be implied based on particular features or arrangements insuch examples, as one of ordinary skill would be able to use theseexamples to derive a wide variety of implementations. Further andalthough some subject matter may have been described in languagespecific to examples of structural features and/or method steps, it isto be understood that the subject matter defined in the appended claimsis not necessarily limited to these described features or acts. Forexample, such functionality can be distributed differently or performedin components other than those identified herein. Rather, the describedfeatures and steps are disclosed as examples of components of systemsand methods within the scope of the appended claims. Moreover, claimlanguage reciting “at least one of” a set indicates that one member ofthe set or multiple members of the set satisfy the claim.

1. A method for modifying audio streams, the method comprising:determining a physical location of a first unmanned aerial vehicle (UAV)within an environment, wherein the physical location of the first UAV isdetermined by one or more sensors monitoring the environment; receivingan audio stream associated with a simulated object in the environment;identifying that an intermediary object is located between the physicallocation of the first UAV and a location of the simulated object withinthe environment; and processing the audio stream before transmitting toan operator output device associated with the first UAV, wherein avolume of the audio stream is degraded based on the identifiedintermediary object.
 2. The method of claim 1, wherein the sensorscomprise at least one of an inertial measurement unit, a GPS receiver, amagnetometer, a barometer, a land-speed sensor, an air-speed sensor, ora celestial navigation sensor.
 3. The method of claim 1, whereinprocessing the audio stream includes applying a low pass filter to theaudio stream based on frequency degrading properties of the identifiedintermediary object.
 4. The method of claim 1, wherein the simulatedobject is represented by a second UAV, and wherein the audio stream isreceived via a microphone associated with the second UAV.
 5. The methodof claim 1, further comprising receiving the audio stream from amicrophone attached to the first UAV, wherein processing the audiostream includes combining the audio stream associated with the simulatedobject with the audio stream from the microphone.
 6. The method of claim1, wherein the simulated object is part of a grouping, and furthercomprising identifying the grouping of the simulated object, whereinprocessing the audio stream is further based on the identified grouping.7. A system for modifying audio streams, the system comprising: one ormore sensors that monitor an environment and detect a physical locationof a first unmanned aerial vehicle (UAV) in the environment; aninterface that receives an audio stream associated with a simulatedobject in the environment; a processor that executes instructions storedin memory, wherein execution of the instructions by the processor:identifies that the intermediary object is located between the physicallocation of the first UAV and a location of the simulated object withinthe environment; and processes the audio stream before transmitting toan operator output device associated with the first UAV, wherein avolume of the audio stream is degraded based on the identifiedintermediary object.
 8. The system of claim 7, wherein the sensorscomprise at least one of an inertial measurement unit, a GPS receiver, amagnetometer, a barometer, a land-speed sensor, an air-speed sensor, ora celestial navigation sensor.
 9. The system of claim 7, wherein theprocessors processes the audio stream by applying a low pass filter tothe audio stream based on frequency degrading properties of theidentified intermediary object.
 10. The system of claim 7, wherein thesimulated object is represented by a second UAV, and wherein theinterface includes a microphone associated with the second UAV thatreceives the audio stream.
 11. The system of claim 7, wherein theinterface further receives an audio stream from a microphone attached tothe first UAV, wherein the processor processes the audio stream bycombining the audio stream associated with the simulated object with theaudio stream from the microphone.
 12. The system of claim 7, wherein thesimulated object is part of a grouping, and wherein the processorfurther identifies the grouping of the simulated object and processesthe audio stream based on the identified grouping.
 13. A non-transitorycomputer-readable storage medium having embodied thereon instructionsexecutable by a processor to perform a method for modifying audiostreams, the method comprising: determining a physical location of afirst unmanned aerial vehicle (UAV) within an environment, wherein thephysical location of the first UAV is detected by one or more sensorsmonitoring the environment; receiving an audio stream associated with asimulated object; identifying that an intermediary object is locatedbetween the physical location of the first UAV and a location of thesimulated object within the environment; and processing the audio streambefore transmitting to an operator output device associated with thefirst UAV, wherein a volume of the audio stream is degraded based on theidentified intermediary object.
 14. The non-transitory computer-readablestorage medium of claim 13, wherein the sensors comprise at least one ofan inertial measurement unit, a GPS receiver, a magnetometer, abarometer, a land-speed sensor, an air-speed sensor, or a celestialnavigation sensor.
 15. The non-transitory computer-readable storagemedium of claim 13, wherein processing the audio stream includesapplying a low pass filter to the audio stream based on frequencydegrading properties of the identified intermediary object.
 16. Thenon-transitory computer-readable storage medium of claim 13, wherein thesimulated object is represented by a second UAV, and wherein receivingthe audio stream comprises capturing audio via a microphone associatedwith the second UAV.
 17. The non-transitory computer-readable storagemedium of claim 13, further comprising instructions executable toreceive an audio stream from a microphone attached to the first UAV,wherein processing the audio stream includes combining the audio streamassociated with the simulated object with the audio stream from themicrophone.
 18. The non-transitory computer-readable storage medium ofclaim 13, wherein the simulated object is part of a grouping, andfurther comprising instructions executable to identify the grouping ofthe simulated object, wherein processing the audio stream is further theidentified grouping of the simulated object.